{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing `music21`\n",
    "\n",
    "`Music21` works only because the code has been extensively tested so that new changes are confirmed not to break existing code.  Time spent testing always slows down coding in the short run, but long experience has shown that it saves coding time in the long run.  Contributions to `music21` without tests will always be rejected, so as developers, it's important that code be well tested.\n",
    "\n",
    "**All functions/methods/etc. need documentation and at least one test.  Oh, and that test must pass.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module-level testing\n",
    "\n",
    "Every `music21` module should have exactly one class called `Test` which should run the tests for the module.  (If the module is getting very big, it is possible to have this class reside in a different module, so long as it is imported into the module that it tests).\n",
    "\n",
    "Make the Test class be a subclass of `unittest` and each of the test methods within it must begin with `test`, for instance `testTupletsEndProperly`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import unittest\n",
    "\n",
    "class Test(unittest.TestCase):\n",
    "\n",
    "    def runTest(self):\n",
    "        pass\n",
    "\n",
    "    def testNotesHaveStemDirection(self):\n",
    "        from music21 import note\n",
    "        n = note.Note('C#')\n",
    "        self.assertTrue(hasattr(n, 'stemDirection'))\n",
    "        n.stemDirection = 'down'\n",
    "        self.assertEqual(n.stemDirection, 'down')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Test` class must not produce any output if the tests succeed.  They must not create new windows, lose the focus from the current window, play music, etc.  They must not rely on modules that are not part of the `music21` ecosystem (i.e., scipy).  \n",
    "\n",
    "If you need to create a second test that generates external output, create a second `unittest.TestCase` called `TestExternal`.  Tests should run quickly: what \"quickly\" means depends on the complexity of what is being contributed; a major new module can add a few seconds to the test suite.  A tiny addition, should add milliseconds at most.  If your test adds 10 seconds to running, that’s 10 seconds that everyone will have to wait when running test. If you need to have a test that takes a lot of time, create a second Test class called `TestSlow`.\n",
    "\n",
    "End your file so that if the module is run directly, it calls these tests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "nbval-ignore-output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.001s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    import music21\n",
    "    music21.mainTest(Test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "instead of `music21.mainTest(Test)` you can write `music21.mainTest(Test, verbose=True)` or while debugging `music21.mainTest(Test, runTest='testNotesHaveStemDirection')` to run a single test.  Or to run a single test, you can pass it from the command line:\n",
    "\n",
    "    python3 note.py testNotesHaveStemDirection\n",
    "\n",
    "You can also call the tests for the module with:\n",
    "\n",
    "    python3 -m music21.note\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Documentation Testing\n",
    "\n",
    "One great feature Python has is the ability to write tests within the documentation itself (\"`doctest`\").  We use a lot of these to demonstrate expected behavior in calling a method or function and also to test that the expected behavior works.  However, doctests do take up space in the documentation, so do not write anything that can't be read (use the module-level testing instead).\n",
    "\n",
    "Doc tests are written like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def flipStemDirection(n):\n",
    "    '''\n",
    "    Takes in a note with a `.stemDirection` of 'up' or 'down'\n",
    "    and transforms it to the other.  Does nothing if the `stemDirection` is not defined.\n",
    "    \n",
    "    Notes without stem direction of `up` or `down` are left alone:\n",
    "    \n",
    "    >>> n = note.Note()\n",
    "    >>> n.stemDirection\n",
    "    'unspecified'\n",
    "    >>> flipStemDirection(n)\n",
    "    >>> n.stemDirection\n",
    "    'unspecified'\n",
    "\n",
    "    Here we flip an `up` to `down` and back again.\n",
    "\n",
    "    >>> n.stemDirection = 'up'\n",
    "    >>> flipStemDirection(n)\n",
    "    >>> n.stemDirection\n",
    "    'down'\n",
    "    >>> flipStemDirection(n)\n",
    "    >>> n.stemDirection\n",
    "    'up'\n",
    "    \n",
    "    Objects without `.stemDirection` raise a Music21Exception\n",
    "    \n",
    "    >>> r = note.Rest()\n",
    "    >>> flipStemDirection(r)\n",
    "    Traceback (most recent call last):\n",
    "    music21.exceptions21.Music21Exception: flipStemDirection only works on Notes\n",
    "    '''\n",
    "    try:\n",
    "        sd = n.stemDirection\n",
    "    except AttributeError:\n",
    "        raise exceptions21.Music21Exception(\"flipStemDirection only works on Notes\")\n",
    "    \n",
    "    if sd == 'up':\n",
    "        n.stemDirection = 'down'\n",
    "    elif sd == 'down':\n",
    "        n.stemDirection = 'up'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally in a doctest, the full path to the function (assuming `from music21 import *`) must be given, so if this function were in a hypothetical `stem.py` file, it would need to be called as `stem.flipStemDirection`.  The reason for this is that we're writing tests and docs at the same time, so we want to document usage as our users (who are suggested to run `from music21 import *`) will need to run the routine.\n",
    "\n",
    "All doctests start with a clean slate of variables, so variables created in a prior doctest will not be available in another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running all tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've made sure your contributions to a module work, you'll want to run all the tests in the system.\n",
    "\n",
    "The simplest way of running all the tests for `music21` is to run:\n",
    "\n",
    "    python3 test/multiprocessTest.py\n",
    "    \n",
    "This will use `(n - 1)` of your n-processor system to run all of the tests and will print a lot of output along the way.  The tests run in reverse-alphabetical order (with some slow tests boosted to the beginning).  All failed tests will appear near the bottom of the output.\n",
    "\n",
    "A few modules are not tested in `multiprocessTest` because they do not play well with multiprocessing or are in obscure subdirectories.  They do not need to be tested on a general basis, but before a major release they should be tested with:\n",
    "\n",
    "    python3 test/singleCoreAll.py\n",
    "    \n",
    "Before a release, this should be run with every version of Python that is supported (and preferably on both Mac and PC).  This can be a pain, so fortunately we have..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Continuous Integration Tests\n",
    "\n",
    "Upon any commit or pull request, continuous integration testing at Travis-CI runs.  See:\n",
    "\n",
    "    https://travis-ci.org/cuthbertLab/music21\n",
    "    \n",
    "and hopefully it's running Green at least for the master branch.  Travis-CI runs the `travisMain()` function of test/singleCoreAll which is the same as the `main()` function except it exits with 0 or -1 or whatever the UNIX convention for failure is (I always forget--that's why I coded it).  Thanks to the great folks at Travis-CI, the risks of accepting pull requests is lower (though not zero, especially in cases where there are changes to external output or to parsing files outside the corpus).  Although the amount of time to run varies depending on the load of travis-ci, keep a look out for changes that significantly slow down the system -- they may indicate an inefficient algorithm.\n",
    "\n",
    "All pull requests **must** pass Travis-CI to be accepted.  It doesn't matter that \"that bug was probably there before\" -- it's time to fix it or to get help fixing it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Coverage Tests\n",
    "\n",
    "One of the systems running on Travis will always run slower than the rest.  This system is running `coverage` to measure how much of the `music21` code is being run.  For the results see:\n",
    "\n",
    "    https://coveralls.io/github/cuthbertLab/music21\n",
    "\n",
    "While 100% code coverage is impossible, increasing the coverage percentage with each new contribution is essential.  We're at 90% code coverage and get the green badge of pride on our README.md file, and we like that!\n",
    "\n",
    "If something should be impossible to trigger, but the code is being maintained for some reason, you can add \"`# pragma: no cover`\" to the line or to the fork, but this is generally a no-no."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the User's Guide\n",
    "\n",
    "Most new contributions will not change the documentation, but occasionally it will.  To make sure that the documentation is correct, run `testDocumentation.py` (under Python 3 only) in the `documentation` directory (outside the `music21` source code directory).  If there are any changes, check to make sure they make sense (maybe there are some new files in the corpus that is changing the count somewhere?) and if they do, rerun the notebook with that documentation (line by line!) and make sure that the new output makes better sense than the previous.\n",
    "\n",
    "This can take some time, so it does not need to be done for every commit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "The `music21` project relies on great tests, both to ensure everything is working properly and also to save time for those who will support your code in the future.  One of the main things we can always use contributors to is writing tests for code that is under tested, so please feel free to grab some code from Coveralls and contribute tests for it.  It may sound like a small contribution, but it's one that will be greatly appreciated."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
